Method for Generating a Training Dataset for Training an Industrial Robot

ABSTRACT

A method for generating a training data set for training an industrial robot which can be trained based on a corresponding training data set, comprising: providing a first imaging information, which describes a first one- or multi-dimensional image of an object which is to be relocated by means of an industrial robot which is to be trained on the basis of the training data set to be generated; processing the first imaging information to generate further imaging information, which describes at least one artificially generated further one- or multi-dimensional image of the object which is to be moved by means of an industrial robot which is to be trained on the basis of the training data set to be generated; and processing the further imaging information to generate a training data set for training an industrial robot which can be trained on the basis of the training data set.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a national stage entry of PCT/EP2021/062343 filedMay 10, 2021, which claims the benefit of German Patent Application No.DE 10 2020 113 277.8 filed May 15, 2020, the entire disclosures of whichare hereby incorporated herein by reference.

FIELD

The disclosure relates to a method for generating a training data setfor training an industrial robot which can be trained on the basis of acorresponding training data set, wherein the industrial robot comprisesat least one handling device, which comprises at least one handlingelement which can be moved in at least one degree of freedom of movementfor handling an object which is to be moved from a first orientationand/or position to a second orientation and/or position.

BACKGROUND

Training data sets for training industrial robots are widely known fromthe field of robotics, more particularly in conjunction with the conceptof machine learning.

Corresponding training data sets aim to train an industrial robot inconjunction with a particular task, such as, for example, apick-and-place or separation task, more particularly such that the dataprovided to said industrial robot within the context of the task, suchas for example capture data relating to objects to be handled, areprocessed in such a manner that an action leading to the resolving ofthe respective task can be executed in a manner as automated aspossible.

Thus corresponding methods for generating training data sets fortraining industrial robots are in principle also known in differentembodiments from the field of robotics.

There is room for improvement in such methods in that, to date, thetraining data sets are sometimes generated in very lengthy and complexprocesses, and therefore ways of generating corresponding training datasets in an easy yet efficient manner are needed. More particularly, auser who wants to train an industrial robot for a particular task shouldbe able to easily execute the generation of corresponding training datasets.

SUMMARY

The object underlying the disclosure is that of specifying an improvedmethod for generating a training data set for an industrial robot, moreparticularly with respect to making it possible to easily but alsoefficiently generate corresponding training data sets.

The object is achieved by a method for generating a training data setfor training an industrial robot according to claim 1. The dependentclaims relate to possible embodiments of the method.

A first aspect of the disclosure relates to a method for generating atraining data set for training an industrial robot which can be trainedon the basis of a corresponding training data set. A correspondingindustrial robot typically comprises at least one handling device, whichcomprises at least one handling element, i.e. for example a grippingelement, suction element etc., for example for handling an object whichis to be relocated from a first spatial orientation and/or position to asecond spatial orientation and/or position. A corresponding handlingdevice of a corresponding industrial robot can optionally be consideredor regarded as an end-effector device. In embodiments, a correspondinghandling element can thus optionally also be considered or regarded asan end-effector element.

A corresponding industrial robot is configured for the automatable orautomated execution of actions to resolve particular assignments ortasks. Corresponding assignments or tasks could be, for example,pick-and-place or relocation or singulation assignments with respect toone or more objects. A corresponding industrial robot can thus be anindustrial robot configured to execute pick-and-place or relocation orseparation assignments with respect to one or more objects.

In embodiments, a corresponding industrial robot can be designed as acollaborative industrial robot (“cobot”) or comprise same. Thus themethod for generating a training data set can be implemented to train acollaborative industrial robot (“cobot”).

A corresponding industrial robot can be a constituent part of anassembly, which in addition to the industrial robot comprises one ormore peripherals. A corresponding peripheral can for example be designedas, or comprise, a feed device for feeding objects, more particularlyobjects which are in a first orientation and/or position, to an actionregion of at least one handling element of the handling device of theindustrial robot. Alternatively or additionally, a correspondingperipheral device can be designed as, or comprise, a removal device forremoving objects, more particularly objects moved by means of theindustrial robot to the second orientation and/or position.

A corresponding assembly or machine can be a packaging machine forpackaging objects, or can be a constituent part of same. A correspondingpackaging machine can for example be set up to move objects, such asfoods, cosmetic items, pharmaceutical items, technical items, from afirst orientation and/or position to a second orientation and/orposition, i.e. for example to a carrier-like receiving device.

A training data set which can be or is generated according to the methodis used for example to generate a model for controlling an industrialrobot in connection with the execution of a particular assignment ortask, on the basis of which model the industrial robot can be controlledto execute the respective assignment or task. As is shown below, data orinformation are provided for this purpose, and are processed in aparticular manner to generate a respective training data set. A trainingdata set generated according to the method thus typically relates to oneor more assignments or tasks to be executed by an industrial robot, theexecution of which the industrial robot is to be “trained” to do. Theprovided data or information for generating a respective training dataset are typically processed using algorithms, which process the provideddata or information for generating a respective training data set, i.e.more particularly with regard to a particular assignment or task whichis to be trained. Corresponding algorithms can for example be set up toidentify patterns and/or regularities in provided data or information,or to extract same from provided data. This can make it possible for“unknown” data to then be processed in a corresponding manner, which canmore particularly permit new patterns and/or regularities to beidentified.

The method for generating a corresponding training data set for trainingan industrial robot comprises the steps described in more detail below:

In a first step of the method, a first imaging information is provided,which describes a first one- or multi-dimensional image of an objectwhich is to be moved by means of an industrial robot which is to betrained on the basis of the training data set to be generated. In thefirst step of the method, a first imaging information is thus providedfor example by means of a suitable provisioning device implemented bymeans of hardware and/or software. The first imaging informationdescribes or relates to a first one- or multi-dimensional image of anobject which is to be moved by means of an industrial robot which is tobe trained on the basis of the training data set to be generated. Thefirst imaging information can describe a one- or multi-dimensional imageof a particular object, for example in a particular orientation and/orposition and/or in front of a particular foreground or background and/orin a particular lighting situation. The object described by the firstimaging information is typically the object the handling of which theindustrial robot is to be trained in on the basis of the training dataset to be generated for the execution of a particular assignment.

As mentioned, the first imaging information can describe for example animage of the respective object in a first spatial orientation and/orposition and/or in a first spatial environment, more particularly infront of a first foreground and/or background, and/or under a firstchemical and/or physical condition, such as, for example, in a firstchemical composition, at a first pressure, having a first moisture,having a first temperature, etc., and/or in a first lighting situationand/or in a first color.

Irrespective of its specific content, the first imaging information istypically a digital information which can be processed using suitabledata or image processing measures implemented by means of hardwareand/or software.

In a second step of the method, the first imaging information isprocessed to generate further imaging information, which describes atleast one artificially generated further one- or multi-dimensional imageof the object which is to be moved by means of an industrial robot whichis to be trained on the basis of the training data set to be generated.In the second step of the method, the first imaging information providedin the first step of the method is thus processed, for example by meansof a suitable data or image processing device implemented by means ofhardware and/or software or is processed by means of cloud computing, inorder to generate a further imaging information. In the second step ofthe method, a further imaging information is thus generated byprocessing the first imaging information. The result of the second stepof the method is thus a further imaging information, which was generatedon the basis of the first imaging information. The further imaginginformation describes or relates to a one- or multi-dimensional image,artificially generated more particularly on the basis of the firstimaging information, of the object to be moved by the industrial robotwhich is to be trained on the basis of the training data set to begenerated. The further imaging information thus typically describes thesame object as the first imaging information; however, in comparisonwith the first imaging information, the object can, in the furtherimaging information, be described in another, artificially generated,one- or multi-dimensional image and/or in another or a further,artificially generated, orientation and/or position and/or in front ofanother or a further, artificially generated, foreground and/orbackgrounds and/or in another or a further, artificially generated,lighting situation. In the further imaging information, the object canthus be described for example in at least one artificially generatedrepresentation, optionally also in an artificially generatedenvironment.

The further imaging information can describe for example an artificiallygenerated image of the object in at least one further spatialorientation and/or position and/or in at least one further spatialenvironment, more particularly in front of at least one furtherbackground, and/or under at least one further chemical and/or physicalcondition, such as, for example, in a further chemical composition, at afurther pressure, having a further moisture, having a furthertemperature, etc., and/or in at least one further lighting situationand/or in at least one further color.

More particularly, the further imaging information can describe anartificially generated image of the respective object in a moreparticularly ordered or unordered arrangement with at least one furtherobject, more particularly at least one further object of the same ordifferent kind or type. Thus an object described in a first imaginginformation can, in the second imaging information, be imaged ordescribed in an arrangement with further objects. In this manner, theinformation content of the further imaging information, whichinformation content is necessary or useful for the generation of atraining data set, can be extended in comparison with the first imaginginformation.

Irrespective of its specific content, the further imaging information istypically a digital information which can be processed using suitabledata or image processing measures implemented by means of hardwareand/or software.

For processing the first imaging information and thus for generating thefurther imaging information, suitable data process measures can thus beapplied, i.e. more particularly suitable imaging processing measures,which enable a corresponding first imaging information to be processedto generate a corresponding further imaging information. Correspondingdata or image processing measures can be implemented using suitable dataor image processing algorithms.

In a third step of the method, the further imaging information isprocessed to generate a training data set for training an industrialrobot which can be trained on the basis of the training data set. In thethird step of the method, the further imaging information generated inthe second step of the method is processed for example by means of theor a suitable data or image processing device implemented by means ofhardware and/or software, in order to generate a training data set onthe basis of which a corresponding industrial robot can be trained. Inthe third step of the method, a training data set is thus generated byprocessing the further imaging information. The result of the third stepof the method is thus a training data set, which was generated on thebasis of the further imaging information.

Important to the method in some embodiments is the generation, whichtakes place in the second step more particularly in an automatable orautomated manner, of the further imaging information, which describesthe respective object in several artificially generated situations, i.e.for example in several different artificially generated orientations orpositions and/or in front of several different artificially generatedforegrounds and/or backgrounds and/or in several different artificiallygenerated lighting situations. The information content contained in thefirst imaging information, i.e. more particularly the informationdescribed in the first imaging information in relation to the respectiveobject, is artificially extended by the further imaging information, asin the further imaging information, information is described in relationto the respective object in at least one artificially generated furtherone- or multi-dimensional image. In some embodiments, this is animportant aspect for the generation of a corresponding training data setand for the training of the respective industrial robot, as the trainingof the industrial robot can thus be carried out on the basis of aplurality of different information, described by the respective furtherimaging information, in relation to the respective object to be moved,although originally, only one (single) imaging information, namely thefirst imaging information, was provided.

Overall, an improved method is thus provided for generating a trainingdata set for training an industrial robot.

As mentioned, the processing of the first imaging information forgenerating the further imaging information can be carried out by meansof at least one image processing measure. More particularly, one or moredigital image processing measures can be applied which, as alsomentioned, can be implemented for example by image processingalgorithms. A corresponding digital image processing measure can includeat least one measure for identifying particular object parameters, moreparticularly geometric-structural object parameters, surface (finish)parameters, optical reflection parameters etc. A corresponding imageprocessing measure can also include at least one measure for identifyingparticular parameters of a foreground and/or background and/orparticular chemical and/or physical conditions and/or particularlighting situations and/or particular colors of the object described inthe first imaging information.

In this context, although in principle also irrespective of it, it mustbe mentioned that a corresponding first imaging information can, inembodiments, contain meta-information or, in embodiments,meta-information can be assigned to a corresponding first imaginginformation. Corresponding items of meta-information can describe orrelate to one or more items of sub-information. Correspondingmeta-information can thus provide details about what is described or“can be seen” in a respective first imaging information. Themeta-information contained in or assigned to the first imaginginformation can be artificially changed and/or artificially replicatedin the further imaging information.

A corresponding sub- or meta-information can thus for example be a typeor class information describing a type or class of at least one object,which type or class has to date also been called a “format” inpick-and-place applications. Thus the type or class of at least oneobject contained in a first imaging information can be used assub-information or as meta-information.

Alternatively or additionally, a corresponding sub- or meta-informationcan for example be an absolute orientation information and/or absoluteposition information describing an absolute orientation and/or absoluteposition of at least one object. An absolute orientation informationand/or absolute position information can be specified in terms ofposition angles and/or world coordinates or contain same. Thus anabsolute orientation and/or position of at least one object contained ina first imaging information can be used as sub-information or asmeta-information.

Alternatively or additionally, a corresponding sub- or meta-informationcan for example be a relative orientation information and/or relativeposition information describing a relative orientation and/or relativeposition of at least one object. A relative orientation informationand/or relative position information can be specified in terms ofposition angles and/or world coordinates or contain same. Thus arelative orientation and/or position of at least one object contained ina first imaging information in relation to at least one further objectcontained in the imaging information can be used as sub-information oras meta-information.

Alternatively or additionally, a corresponding sub- or meta-informationcan for example be a foreground information and/or a backgroundinformation describing a foreground and/or background of at least oneobject. Thus the foreground and/or background of at least one objectcontained in a first imaging information can be used as sub-informationor as meta-information.

Alternatively or additionally, a corresponding sub- or meta-informationcan for example be a lighting information describing a lightingsituation or the lighting conditions of at least one object. Thus thelighting situation or lighting conditions of at least one objectcontained in a first imaging information can be used as sub-informationor as meta-information.

Alternatively or additionally, a corresponding sub- or meta-informationcan for example be a driving information describing a driving movementrequired more particularly from an ACTUAL position and/or ACTUALorientation or a driving vector of a handling element of the handlingdevice of the industrial robot for driving at least one object containedin a first imaging information. Thus a driving movement required moreparticularly from and ACTUAL position and/or ACTUAL orientation or adriving vector of a handling element of the handling device of theindustrial robot can be detected and used as sub-information ormeta-information.

Alternatively or additionally, a corresponding sub- or meta-informationcan for example be a dimension information describing a at least onegeometric-structural dimension of at least one object. Thus at least onedimension of at least one object contained in a first imaginginformation can be used as sub-information or as meta-information.

Alternatively or additionally, a corresponding sub- or meta-informationcan for example be a shape information describing a at least onegeometric-structural shape (three-dimensional shape) of at least oneobject. Thus at least one shape of at least one object contained in afirst imaging information can be used as sub-information or asmeta-information.

Alternatively or additionally, a corresponding sub- or meta-informationcan for example be a color information describing a color of at leastone object. Thus at least one color of at least one object contained ina first imaging information can be used as sub-information or asmeta-information.

Alternatively or additionally, a corresponding sub- or meta-informationcan be a product name information describing, or described by,alphanumeric and/or graphical elements, such as for example a productname, of at least one object, optionally also of a packaging of anobject. Thus the alphanumeric and/or graphical elements of at least oneobject contained in a first imaging information can be used assub-information or as meta-information.

Alternatively or additionally, a corresponding sub- or meta-informationcan for example be a surface information describing a surface, moreparticularly a surface finish, i.e. more particularly the opticalsurface properties, such as for example a degree of shine or reflection,of at least one object. Thus the surface, more particularly the surfacefinish, of at least one object contained in a first imaging informationcan be used as sub-information or as meta-information.

Alternatively or additionally, a corresponding sub- or meta-informationcan for example be a mass and/or volume information describing a mass,more particularly a center of mass, and/or the volume, more particularlya center of volume, of at least one object. Thus the mass, moreparticularly a center of mass, and/or the volume, more particularly acenter of volume, of at least one object contained in a first imaginginformation can be used as sub-information or as meta-information.

As mentioned, the digital image processing measure for generating thefurther imaging information can be carried out by means of cloudcomputing. Alternatively or additionally, it is possible to implementthe digital image processing measure by means of at least one deeplearning measure, more particularly with the involvement of at least oneone- or multi-layer artificial neural network.

The generation of the training data set carried out in the third step ofthe method can optionally be carried out by means of cloud computing.Thus a corresponding training data set can in principle be carried outcentrally or decentrally. A central generation of a correspondingtraining data set can for example be achieved if the training data setis carried out by means of a central or local data processing device,such as a local computer, smartphone, tablet, etc. A decentralgeneration of a corresponding training data set can for example beachieved if the training data set is carried out by means of a decentralor global data processing device such as a server connected to asuperordinate data or communication network such as for example anintranet or the internet.

The first imaging information for example can be, or can be provided as,a digital image information of the object generated by an image captureor camera device implemented more particularly by means of hardwareand/or software. A first imaging information can thus be for example aphotograph or a video of a respective object or of an image of arespective object. The first imaging information can thus for example begenerated by an image capture or camera device optionally integrated ina user-side (mobile) terminal such as for example a smartphone, tablet,laptop, etc.

Alternatively or additionally, the first imaging information can be, orcan be provided as, a digital design information of a respective objectgenerated by means of a design program device, more particularlyimplemented by means of hardware and/or software. A first imaginginformation can for example be a CAD file, STL file, etc. of arespective object. The first imaging information can thus be generatedfor example by a computer-based design program, such as for example aCAD program.

Alternatively or additionally, the first imaging information can be, orcan be provided as, an electronic document including an image of arespective object or a corresponding file. A first imaging informationcan for example be an editable or non-editable document which includes arespective object. The first imaging information can thus be a pdf file,a presentation file, a word processing file, a web page etc.

In embodiments, the method can comprise a step of transmitting thefurther imaging information to an industrial robot to be trained and/orto a cloud computing device. Thus the further imaging information can betransmitted via a suitable data or communications connection to anindustrial robot to be trained and/or to a cloud computing device. Acorresponding data or communications connection can be or comprise awired or wireless data or communications connection via one or more dataor communication networks. A corresponding data or communicationsconnection can be encrypted or unencrypted.

A second aspect of the disclosure relates to a method for training anindustrial robot which can be trained on the basis of a correspondingtraining data set, wherein the industrial robot comprises at least onehandling device, which for example comprises at least one handlingelement which can be moved in at least one degree of freedom of movementfor handling an object which is to be moved from a first orientationand/or position to a second orientation and/or position. The methodcomprises the following steps: Providing a training data set which wasgenerated according to a method according to the first aspect of thedisclosure, and training the industrial robot on the basis of theprovided training data set. Embodiments in connection with the methodaccording to the first aspect of the disclosure apply accordingly to themethod according to the second aspect of the disclosure, and vice versa.

The training of the industrial robot can be implemented or carried outby means of at least one measure for machine learning. The at least onemeasure for machine learning typically includes the processing of acorresponding training data set. The at least one measure for machinelearning can more particularly be implemented or carried out by means ofat least one deep learning measure, more particularly with theinvolvement of at least one one- or multi-layer artificial neuralnetwork. A respective artificial neural network can thus have one ormore intermediate layers implemented between an input layer and anoutput layer. More particularly, neural networks having a plurality ofcorresponding intermediate layers can be used, as in this manner, bettertraining results can regularly be achieved.

The training of the industrial robot can be performed or implemented bymeans of cloud computing. The industrial robot to be trained can thus beconnected, via a suitable data or communications connection, to a cloudcomputing device for example in the form of a corresponding ITinfrastructure or a computer network, via which the training of theindustrial robot can be carried out. Thus a correspondingly generatedtraining data set can be processed via cloud computing for training theindustrial robot. The industrial robot therefore does not necessarilyhave to be configured by means of hardware and/or software to “trainitself”. Rather, the training of the industrial robot can take place viaa corresponding IT infrastructure or a corresponding computer network,with which the industrial robot communicates via a data orcommunications connection.

The training of the industrial robot can contain at least onesimulation-executed action and/or at least one actually executed actionof the handling device of the industrial robot. More particularly, it isconceivable that trained actions of the handling device are firstsimulated at least once before being actually executed.

Within the context of training of the industrial robot, a control dataset can be generated on the basis of the provided training data set inorder to control the operation of the industrial robot. The control dataset can contain control information for executing a particularassignment or task. During further training of the industrial robot, thecontrol data set can be updated or changed.

A third aspect of the disclosure relates to a method for controlling theoperation of an industrial robot, wherein the industrial robot comprisesat least one handling device, which for example comprises at least onehandling element which can be moved in at least one degree of freedom ofmovement for handling an object which is to be moved from a firstorientation and/or position to a second orientation and/or position. Themethod comprises the following steps: Providing a control data set forcontrolling the operation of the industrial robot, wherein the controldata set was generated by means of a method according to the firstaspect of the disclosure, or the control data was generated on the basisof a method according to the second aspect of the disclosure, andcontrolling the operation of the industrial robot on the basis of theprovided control data set. Embodiments in connection with the methodaccording to the first aspect of the disclosure and the method accordingto the second aspect of the disclosure apply accordingly to the methodaccording to the third aspect of the disclosure, and vice versa.

A fourth aspect of the disclosure relates to an industrial robot, moreparticularly a collaborative industrial robot (“cobot”), comprising atleast one handling device, which for example comprises at least onehandling element which can be moved in at least one degree of freedom ofmovement for handling an object which is to be moved from a firstorientation and/or position to a second orientation and/or position. Theindustrial robot is trained on the basis of a training data setgenerated according to a method according to the first aspect of thedisclosure, or is trained on the basis of a method according to thesecond aspect of the disclosure, and or is controllable or controlled onthe basis of a method according to the third aspect of the disclosure.Embodiments in connection with the method according to the first aspectof the disclosure, the method according to the second aspect of thedisclosure, the method according to the third aspect of the disclosureapply similarly to the industrial robot according to the fourth aspectof the disclosure, and vice versa.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure is explained in more detail on the basis of the exemplaryembodiments in the drawings.

FIG. 1 provides a schematic representation of an assembly for movingobjects from a first orientation and/or position to a second orientationand/or position according to an exemplary embodiment;

FIG. 2 provides a schematic representation of a first imaginginformation according to an exemplary embodiment;

FIG. 3 provides a schematic representation of a further imaginginformation according to an exemplary embodiment; and

FIG. 4 provides a block diagram for representing a method for generatinga training data set for training an industrial robot according to anexemplary embodiment.

DETAILED DESCRIPTION

FIG. 1 shows a schematic representation of an assembly 1 for relocatingobjects 2 from a first orientation and/or position to a secondorientation and/or position according to an exemplary embodiment in atop view. The assembly 1 can also be described or regarded as a machine.

The assembly 1 comprises an industrial robot 3 designed for example as acollaborative industrial robot (“cobot”) and several peripherals. In theembodiment, the peripherals are: a feed device 4 for example designed asa feed belt for feeding objects 2, more particularly objects 2 in afirst orientation and/or position, to an action region 5 of an endeffector element or handling element 6, for example designed as agripping or suction element, of an end effector or handling device 7 ofthe industrial robot 3; and a removal device 9, for example designed asa removal belt, for removing objects 2, more particularly objects 2moved to a second orientation and/or position by means of the industrialrobot.

The dashed lines indicate that the assembly 1 can also comprise severalcorresponding peripherals and several corresponding end effector orhandling devices 7 in addition to the associated end effector orhandling element 6.

The assembly 1 also comprises a control device 8 implemented by means ofhardware and/or software, which is set up to control the operation ofthe industrial robot 3. The control device 8 shown in FIG. 1 purely byway of example as a structural constituent part of the industrial robot3 is thus set up to generate control data sets or control information,on the basis of which the operation of the industrial robot 3 iscontrolled in order to execute particular assignments or tasks.Corresponding assignments or tasks could be, for example, pick-and-placeor move or simulating assignments with respect to one or more objects 2.

The control data sets or control information on which operation of theindustrial robot 3 is based were generated within the context oftraining of the industrial robot 3. The training of the industrial robot3 is carried out on the basis of a training data set.

The training of the industrial robot 3 can be implemented or carried outby means of at least one measure for machine learning. The at least onemeasure for machine learning includes the processing of a correspondingtraining data set. The at least one measure for machine learning canmore particularly be implemented or carried out by means of at least onedeep learning measure, more particularly with the involvement of atleast one one- or multi-layer artificial neural network. A respectiveartificial neural network can thus have one or more intermediate layersimplemented between an input layer and an output layer.

The training of the industrial robot 3 can be carried out or implementedby means of cloud computing. The industrial robot 3 can thus beconnected, via a suitable data or communications connection, to a cloudcomputing device 9, i.e. a corresponding IT infrastructure or a computernetwork, via which the training of the industrial robot 3 can be carriedout. Thus a training data set can be processed via cloud computing fortraining the industrial robot 3. The industrial robot 3 therefore doesnot necessarily have to be configured by means of hardware and/orsoftware to “train itself”. Rather, the training of the industrial robot3 can take place via a corresponding IT infrastructure or acorresponding computer network, with which the industrial robot 3communicates via a data or communications connection.

The training of the industrial robot 3 can contain at least onesimulation-executed action and/or at least one actually executed actionof the handling device 7. More particularly, it is conceivable thattrained actions of the handling device 7 are first simulated at leastonce before being actually executed.

As mentioned, within the context of training of the industrial robot 3,a control data set can be generated on the basis of the training dataset in order to control the operation of the industrial robot 3. Thecontrol data set can contain control information for executing aparticular assignment or task. During further training of the industrialrobot, the control data set can be updated or changed.

An exemplary embodiment of a method for generating a training data setfor training the industrial robot 3 is described in more detail belowwith reference to FIGS. 2-4 .

A corresponding training data set is typically used to generate a modelfor controlling an industrial robot 3 in connection with the executionof a particular assignment or task, on the basis of which the industrialrobot 3 can be controlled to execute the respective assignment or task.Data or information are provided for this purpose, and are processed ina particular manner to generate a respective training data set. Acorresponding training data set thus typically relates to one or moreassignments or tasks to be executed by an industrial robot, theexecution of which the industrial robot is to be “trained” or “learned”to do. The provided data or information for generating a respectivetraining data set are typically processed using algorithms, whichprocess the provided data or information for generating a respectivetraining data set, i.e. more particularly with regard to a particularassignment or task which is to be trained. Corresponding algorithms canfor example be set up to identify patterns and/or regularities inprovided data or information, or to extract same from provided data.This can make it possible for “unknown” data to then be processed in acorresponding manner, which can more particularly permit new patternsand/or regularities to be identified.

In a first step Si of the method, a first imaging information All isprovided for example by means of a suitable provisioning deviceimplemented by means of hardware and/or software. The first imaginginformation All describes or relates to a first one- ormulti-dimensional image of an object 2 which is to be moved by means ofan industrial robot 3 which is to be trained on the basis of thetraining data set to be generated. The first imaging information Allshown in an exemplary embodiment in FIG. 2 can describe a one- ormulti-dimensional image of a particular object 2, for example in aparticular orientation and/or position and/or in front of a particularforeground or background and/or in a particular lighting situation. Theobject described by the first imaging information is typically theobject the handling of which the industrial robot is to be trained in onthe basis of the training data set to be generated for the execution ofa particular assignment.

In the exemplary embodiment shown in FIG. 2 , the first imaginginformation I1 describes a multi-dimensional image of an object 2 imagedin front of a neutral background. The object 2 has an elongate geometrypurely by way of example. The object 2 can specifically be for example achocolate bar to be packaged in a box-like or carrier-like packaging forreceiving a plurality of chocolate bars.

The first imaging information All thus describes an image of therespective object 2 in a first spatial orientation and/or positionand/or in a first spatial environment, more particularly in front of afirst foreground and/or background, and/or under a first chemical and/orphysical condition, such as, for example, in a first chemicalcomposition, at a first pressure, having a first moisture, having afirst temperature, etc., and/or in a first lighting situation and/or ina first color.

Irrespective of its specific content, the first imaging information AI1is typically a digital information which can be processed using suitabledata or image processing measures implemented by means of hardwareand/or software.

In a second step S2 of the method, the first imaging information AI1provided in the first step S1 of the method is processed, for example bymeans of a suitable data or image processing device implemented by meansof hardware and/or software or is processed by means of cloud computing,in order to generate a further imaging information AI2. The result ofthe second step S2 of the method is thus a further imaging informationAI2, which was generated on the basis of the first imaging informationAI1. The further imaging information AI2 shown in an embodiment in FIG.3 describes or relates to a one- or multi-dimensional image,artificially generated more particularly on the basis of the firstimaging information AI1, of the object 2 to be moved by the industrialrobot 2 which is to be trained on the basis of the training data set tobe generated. The further imaging information AI2 thus typicallydescribes the same object 2 as the first imaging information AI1;however, in comparison with the first imaging information AI1, theobject 2 can, in the further imaging information AI2, be described inanother, artificially generated, one- or multi-dimensional image and/orin another or a further, artificially generated, orientation and/orposition and/or in front of another or a further, artificiallygenerated, foreground and/or backgrounds and/or in another or further,artificially generated, lighting situation. In the further imaginginformation AI2, the object 2 can thus be described for example in atleast one artificially generated representation, optionally also in anartificially generated environment.

The further imaging information AI2 thus describes an artificiallygenerated image of the object 2 in at least one further spatialorientation and/or position and/or in at least one further spatialenvironment, more particularly in front of at least one furtherbackground, and/or under at least one further chemical and/or physicalcondition, such as, for example, in a further chemical composition, at afurther pressure, having a further moisture, having a furthertemperature, etc., and/or in at least one further lighting situationand/or in at least one further color.

As indicated in FIG. 3 , the further imaging information AI2 candescribe an artificially generated image of the respective object 2 in amore particularly ordered or unordered arrangement with at least onefurther object 2′ of the same or different kind or type. Thus the object2 described in a first imaging information AI1 can, in the secondimaging information AI2, be respectively imaged or described in anartificially generated arrangement with further objects 2′.

Irrespective of its specific content, the further imaging informationAI2 is typically a digital information which can be processed usingsuitable data or image processing measures implemented by means ofhardware and/or software.

For processing the first imaging information AI1 and thus for generatingthe further imaging information AI2, suitable data process measures canthus be applied, i.e. more particularly suitable imaging processingmeasures, which enable a corresponding first imaging information AI1 tobe processed to generate a corresponding further imaging informationAI2. Corresponding data or image processing measures can be implementedusing suitable data or image processing algorithms.

In a third step S3 of the method, the further imaging information AI2generated in the second step S2 of the method is processed for exampleby means of the or a suitable data or image processing deviceimplemented by means of hardware and/or software, in order to generate atraining data set TDS on the basis of which a corresponding industrialrobot 3 can be trained. In the third step S3 of the method, a trainingdata set TDS is thus generated by processing the further imaginginformation AI2. The result of the third step S3 of the method is thus atraining data set TDS, which was generated on the basis of the furtherimaging information AI2.

Important to the method in some embodiments is the generation, whichtakes place in the second step S2 more particularly in an automatable orautomated manner, of the further imaging information AI2, whichdescribes the respective object 2 in several artificially generatedsituations, i.e. for example in several different artificially generatedorientations or positions and/or in front of several differentartificially generated foregrounds and/or backgrounds and/or in severaldifferent artificially generated lighting situations. The informationcontent contained in the first imaging information AI1, i.e. moreparticularly the information described in the first imaging informationAI1 in relation to the respective object 2, is artificially extended bythe further imaging information AI2, as in the further imaginginformation AI2, information is described in relation to the respectiveobject 2 in at least one artificially generated further one- ormulti-dimensional image. In some embodiments, this is an importantaspect for the generation of a training data set TDS and for thetraining of the respective industrial robot 3, as the training of theindustrial robot 3 can be carried out on the basis of a plurality ofdifferent information, described by the respective further imaginginformation AI2, in relation to the respective object 2 to be relocated,although originally, only one (single) imaging information AI1, namelythe first imaging information AI1, was provided.

As mentioned, the processing of the first imaging information AI1 forgenerating the further imaging information AI2 can be carried out bymeans of at least one image processing measure. More particularly, oneor more digital image processing measures can be applied which, as alsomentioned, can be implemented for example by image processingalgorithms. A corresponding digital image processing measure can containat least one measure for identifying particular object parameters, moreparticularly geometric-structural object parameters, surface (finish)parameters, optical reflection parameters etc. A corresponding imageprocessing measure can also contain at least one measure for identifyingparticular parameters of a foreground and/or background and/orparticular chemical and/or physical conditions and/or particularlighting situations and/or particular colors of the object described inthe first imaging information AI1.

A first imaging information AI1 can contain meta-information ormeta-information can be assigned to a first imaging information AI1.Corresponding meta-information can describe or relate to one or morepieces of sub-information described in the first imaging information AILThe meta-information can thus typically provide details about what isdescribed or “can be seen” in a respective first imaging information AILThe meta-information contained in or assigned to the first imaginginformation AI1 can be artificially changed and/or artificiallyreplicated in the further imaging information AI2.

A corresponding sub- or meta-information can thus for example be a typeor class information describing a type or class of at least one object2, which type or class has to date also been called a “format” inpick-and-place applications. Thus the type or class of at least oneobject 2 contained in a first imaging information can be used assub-information or as meta-information.

Alternatively or additionally, a corresponding sub- or meta-informationcan for example be an absolute orientation information and/or absoluteposition information describing an absolute orientation and/or absoluteposition of at least one object 2. An absolute orientation informationand/or absolute position information can be specified in terms ofposition angles and/or world coordinates or contain same. Thus anabsolute orientation and/or position of at least one object 2 containedin a first imaging information AI1 can be used as sub-information or asmeta-information.

Alternatively or additionally, a corresponding sub- or meta-informationcan for example be a relative orientation information and/or relativeposition information describing a relative orientation and/or relativeposition of at least one object 2. A relative orientation informationand/or relative position information can be specified in terms ofposition angles and/or world coordinates or contain same. Thus arelative orientation and/or position of at least one object 2 containedin a first imaging information AI1 in relation to at least one furtherobject contained in the first imaging information AI1 can be used assub-information or as meta-information.

Alternatively or additionally, a corresponding sub- or meta-informationcan for example be a foreground information and/or a backgroundinformation describing a foreground and/or background of at least oneobject 2. Thus the foreground and/or background of at least one object 2contained in a first imaging information AI1 can be used assub-information or as meta-information.

Alternatively or additionally, a corresponding sub- or meta-informationcan for example be a lighting information describing a lightingsituation or the lighting conditions of at least one object 2. Thus thelighting situation or lighting conditions of at least one object 2contained in a first imaging information AI1 can be used assub-information or as meta-information.

Alternatively or additionally, a corresponding sub- or meta-informationcan for example be a driving information describing a driving movementrequired more particularly from an ACTUAL position and/or ACTUALorientation or a driving vector of a handling element 6 of the handlingdevice 7 of the industrial robot 3 for driving at least one object 2contained in a first imaging information AI1 Thus a driving movementrequired more particularly from and ACTUAL position and/or ACTUALorientation or a driving vector of a handling element 6 of the handlingdevice 7 of the industrial robot 3 can be detected and used assub-information or meta-information.

Alternatively or additionally, a corresponding sub- or meta-informationcan for example be a measurement information describing a at least onegeometric-structural measurement of at least one object 2. Thus at leastone measurement of at least one object 2 contained in a first imaginginformation AI1 can be used as sub-information or as meta-information.

Alternatively or additionally, a corresponding sub- or meta-informationcan for example be a shape information describing a at least onegeometric-structural shape (three-dimensional shape) of at least oneobject 2. Thus at least one shape of at least one object 2 contained ina first imaging information AI1 can be used as sub-information or asmeta-information.

Alternatively or additionally, a corresponding sub- or meta-informationcan for example be a color information describing a color of at leastone object 2. Thus at least one color of at least one object 2 containedin a first imaging information AI1 can be used as sub-information or asmeta-information.

Alternatively or additionally, a corresponding sub- or meta-informationcan be a product name information describing, or described by,alphanumeric and/or graphical elements, such as for example a productname, of at least one object 2, optionally also of a packaging of anobject 2. Thus the alphanumeric and/or graphical elements of at leastone object 2 contained in a first imaging information can be used assub-information or as meta-information.

Alternatively or additionally, a corresponding sub- or meta-informationcan for example be a surface information describing a surface, moreparticularly a surface finish, i.e. more particularly the opticalsurface properties, such as for example a degree of shine or reflection,of at least one object 2. Thus the surface, more particularly thesurface finish, of at least one object 2 contained in a first imaginginformation AI1 can be used as sub-information or as meta-information.

Alternatively or additionally, a corresponding sub- or meta-informationcan for example be a mass and/or volume information describing a mass,more particularly a center of mass, and/or the volume, more particularlya center of volume, of at least one object 2. Thus the mass, moreparticularly a center of mass, and/or the volume, more particularly acenter of volume, of at least one object 2 contained in a first imaginginformation can be used as sub-information or as meta-information.

As mentioned, the digital image processing measure for generating thefurther imaging information AI2 can be carried out by means of cloudcomputing. Alternatively or additionally, it is possible to implementthe digital image processing measure by means of at least one deeplearning measure, more particularly with the involvement of at least oneone- or multi-layer artificial neural network.

The generation of the training data set TDS carried out in the thirdstep S3 of the method can optionally be carried out by means of cloudcomputing. Thus a corresponding training data set TDS can in principlebe carried out centrally or decentrally. A central generation of acorresponding training data set TDS can for example be achieved if thetraining data set TDS is carried out by means of a central or local dataprocessing device, such as a local computer, smartphone, tablet, etc. Adecentral generation of a corresponding training data set TDS can forexample be achieved if the training data set TDS is carried out by meansof a decentral or global data processing device such as a serverconnected to a superordinate data or communication network such as forexample an intranet or the internet.

The first imaging information AI1 for example can be, or can be providedas, a digital image information of the respective object 2 generated byan image capture or camera device implemented more particularly by meansof hardware and/or software. A first imaging information AI1 can thus befor example a photograph or a video of a respective object 2 or of animage of the object 2. The first imaging information AI1 can thus forexample be generated by an image capture or camera device optionallyintegrated in a user-side (mobile) terminal such as for example asmartphone, tablet, laptop, etc.

Alternatively or additionally, the first imaging information AI1 can be,or can be provided as, a digital design information of the respectiveobject generated by means of a design program device, more particularlyimplemented by means of hardware and/or software. A first imaginginformation AI1 can for example be a CAD file, STL file, etc. of theobject 2. The first imaging information AI1 can thus be generated forexample by a computer-based design program, such as for example a CADprogram.

Alternatively or additionally, the first imaging information AI1 can be,or can be provided as, an electronic document containing an image of therespective object 2 or a corresponding file. A first imaging informationAI1 can for example be an editable or non-editable document whichincludes the respective object 2. The first imaging information AI1 canthus be a pdf file, a presentation file, a word processing file, a webpage etc.

The method can comprise a step of transmitting the further imaginginformation AI2 to an industrial robot 3 to be trained and/or to a cloudcomputing device. Thus the further imaging information can betransmitted via a suitable data or communications connection to anindustrial robot 3 to be trained and/or to a cloud computing device. Acorresponding data or communications connection can be or comprise awired or wireless data or communications connection via one or more dataor communication networks. A corresponding data or communicationsconnection can be encrypted or unencrypted.

1. A method for generating a training data set for training anindustrial robot which can be trained on the basis of a correspondingtraining data set, wherein the industrial robot comprises at least onehandling device which comprises at least one handling element which canbe moved in at least one degree of freedom of movement for handling anobject which is to be relocated from a first orientation and/or positionto a second orientation and/or position, the method comprising thefollowing steps: providing a first imaging information, which describesa first one- or multi-dimensional image of an object which is to bemoved by means of an industrial robot which is to be trained on thebasis of the training data set to be generated, processing the firstimaging information to generate further imaging information, whichdescribes at least one artificially generated further one- ormulti-dimensional image of the object which is to be moved by means ofan industrial robot which is to be trained on the basis of the trainingdata set to be generated, and processing the further imaging informationto generate a training data set for training an industrial robot whichcan be trained on the basis of the training data set.
 2. The methodaccording to claim 1, wherein the first imaging information describes animage of the object in a first spatial orientation and/or positionand/or in a first spatial environment, in front of a first background,and/or under a first chemical and/or physical condition and/or in afirst lighting situation and/or in a first color.
 3. The methodaccording to claim 1, wherein the further imaging information describesan artificially generated image of the object in at least one furtherspatial orientation and/or position and/or in at least one furtherspatial environment, in front of a further background, and/or under atleast one further chemical and/or physical condition and/or in at leastone further lighting situation and/or in at least one further color. 4.The method according to claim 1, wherein the further imaging informationdescribes an artificially generated image of the object in a moreparticularly ordered or unordered arrangement with at least one furtherobject of the same or different kind.
 5. The method according to claim1, wherein the processing of the first imaging information forgenerating the further imaging information is carried out by means of adigital image processing measure.
 6. The method according to claim 5,wherein the digital image processing measure includes at least onemeasure for detecting particular object parameters, more particularlygeometric-structural object parameters, surface (finish) parameters. 7.The method according to claim 5, wherein the digital image processingmeasure is carried out by means of cloud computing.
 8. The methodaccording to claim 5, wherein the digital image processing measure isimplemented by means of at least one deep learning measure, with theinvolvement of at least one artificial neural network.
 9. The methodaccording to claim 1, wherein the first imaging information can beprovided as a digital image information of the object generated by meansof a camera device implemented by means of hardware and/or software, oras a digital design information of the object generated by means of adesign program device implemented by means of hardware and/or software,or as an electronic document including an image of the object.
 10. Themethod according to claim 1, comprising the transmission of the furtherimaging information to an industrial robot to be trained and/or to acloud computing device.
 11. A method for training an industrial robotwhich can be trained on the basis of a training data set, wherein theindustrial robot comprises at least one handling device which comprisesat least one handling element which can be moved in at least one degreeof freedom of movement for handling an object which is to be relocatedfrom a first orientation and/or position to a second orientation and/orposition, comprising the following steps: providing a training data set,which was generated according to the method according to claim 1, andtraining the industrial robot on the basis of the provided training dataset.
 12. Method The method according to claim 11, wherein the trainingof the industrial robot includes at least one actually executed and/orsimulation-executed action of the handling device of the industrialrobot.
 13. The method according to claim 11, wherein within the contextof training of the industrial robot on the basis of the providedtraining data set, a control data set for controlling the operation ofthe industrial robot is generated.
 14. A method for controlling theoperation of an industrial robot, wherein the industrial robot comprisesat least one handling device which comprises at least one handlingelement which can be moved in at least one degree of freedom of movementfor handling an object which is to be relocated from a first orientationand/or position to a second orientation and/or position, the methodcomprising the following steps: providing a control data set forcontrolling the operation of the industrial robot, wherein the controldata set was generated on the basis of a training data set generatedaccording to the method according to one of claim 1, and controlling theindustrial robot on the basis of the provided control data set.
 15. Anindustrial robot, comprising at least one handling device whichcomprises at least one handling element which can be moved in at leastone degree of freedom of movement for handling an object which is to berelocated from a first orientation and/or position to a secondorientation and/or position, wherein the industrial robot is trained onthe basis of a training data set generated according to the methodaccording to claim 1.